Classification accuracy comparison: hypothesis tests and the use of confidence intervals in evaluations of difference, equivalence and non-inferiority

The comparison of classification accuracy statements has generally been based upon tests of difference or inequality when other scenarios and approaches may be more appropriate. Procedures for evaluating two scenarios with interest focused on the similarity in accuracy values, non-inferiority and equivalence, are outlined following a discussion of tests of difference (inequality). It is also suggested that the confidence interval of the difference in classification accuracy may be used as well as or instead of conventional hypothesis testing to reveal more information about the disparity in the classification accuracy values compared

An assessment of citizen contributed ground reference data for land cover map accuracy assessment

It is now widely accepted that an accuracy assessment should be part of a thematic mapping programme. Authoritative good or best practices for accuracy assessment have been defined but are often impractical to implement. Key reasons for this situation are linked to the ground reference data used in the accuracy assessment. Typically, it is a challenge to acquire a large sample of high quality reference cases in accordance to desired sampling designs specified as conforming to good practice and the data collected are normally to some degree imperfect limiting their value to an accuracy assessment which implicitly assumes the use of a gold standard reference. Citizen sensors have great potential to aid aspects of accuracy assessment. In particular, they may be able to act as a source of ground reference data that may, for example, reduce sample size problems but concerns with data quality remain. The relative strengths and limitations of citizen contributed data for accuracy assessment are reviewed in the context of the authoritative good practices defined for studies of land cover by remote sensing. The article will highlight some of the ways that citizen contributed data have been used in accuracy assessment as well as some of the problems that require further attention, and indicate some of the potential ways forward in the future

Classification accuracy comparison: hypothesis tests and the use of confidence intervals in evaluations of difference, equivalence and non-inferiority

The comparison of classification accuracy statements has generally been based upon tests of difference or inequality when other scenarios and approaches may be more appropriate. Procedures for evaluating two scenarios with interest focused on the similarity in accuracy values, non-inferiority and equivalence, are outlined following a discussion of tests of difference (inequality). It is also suggested that the confidence interval of the difference in classification accuracy may be used as well as or instead of conventional hypothesis testing to reveal more information about the disparity in the classification accuracy values compared

Super-resolution land cover mapping by deep learning

Super-resolution mapping (SRM) is a technique to estimate a fine spatial resolution land cover map from coarse spatial resolution fractional proportion images. SRM is often based explicitly on the use of a spatial pattern model that represents the land cover mosaic at the fine spatial resolution. Recently developed deep learning methods have considerable potential as an alternative approach for SRM, based on learning the spatial pattern of land cover from existing fine resolution data such as land cover maps. This letter proposes a deep learning-based SRM algorithm (DeepSRM). A deep convolutional neural network was first trained to estimate a fine resolution indicator image for each class from the coarse resolution fractional image, and all indicator maps were then combined to create the final fine resolution land cover map based on the maximal value strategy. The results of an experiment undertaken with simulated images show that DeepSRM was superior to conventional hard classification and a suite of popular SRM algorithms, yielding the most accurate land cover representation. Consequently, methods such as DeepSRM may help exploit the potential of remote sensing as a source of accurate land cover information

Key issues in rigorous accuracy assessment of land cover products

© 2019 Accuracy assessment and land cover mapping have been inexorably linked throughout the first 50 years of publication of Remote Sensing of Environment. The earliest developers of land-cover maps recognized the importance of evaluating the quality of their maps, and the methods and reporting format of these early accuracy assessments included features that would be familiar to practitioners today. Specifically, practitioners have consistently recognized the importance of obtaining high quality reference data to which the map is compared, the need for sampling to collect these reference data, and the role of an error matrix and accuracy measures derived from the error matrix to summarize the accuracy information. Over the past half century these techniques have undergone refinements to place accuracy assessment on a more scientifically credible footing. We describe the current status of accuracy assessment that has emerged from nearly 50 years of practice and identify opportunities for future advances. The article is organized by the three major components of accuracy assessment, the sampling design, response design, and analysis, focusing on good practice methodology that contributes to a rigorous, informative, and honest assessment. The long history of research and applications underlying the current practice of accuracy assessment has advanced the field to a mature state. However, documentation of accuracy assessment methods needs to be improved to enhance reproducibility and transparency, and improved methods are required to address new challenges created by advanced technology that has expanded the capacity to map land cover extensively in space and intensively in time

Mapping the species richness and composition of tropical forests from remotely sensed data with neural networks

The understanding and management of biodiversity is often limited by a lack of data. Remote sensing has considerable potential as a source of data on biodiversity at spatial and temporal scales appropriate for biodiversity management. To-date, most remote sensing studies have focused on only one aspect of biodiversity, species richness, and have generally used conventional image analysis techniques that may not fully exploit the data's information content. Here, we report on a study that aimed to estimate biodiversity more fully from remotely sensed data with the aid of neural networks. Two neural network models, feedforward networks to estimate basic indices of biodiversity and Kohonen networks to provide information on species composition, were used. Biodiversity indices of species richness and evenness derived from the remotely sensed data were strongly correlated with those derived from field survey. For example, the predicted tree species richness was significantly correlated with that observed in the field (r=0.69, significant at the 95% level of confidence). In addition, there was a high degree of correspondence (?83%) between the partitioning of the outputs from Kohonen networks applied to tree species and remotely sensed data sets that indicated the potential to map species composition. Combining the outputs of the two sets of neural network based analyses enabled a map of biodiversity to be produce

Harshness in image classification accuracy assessment.

Abstract Thematic mapping via a classification analysis is one of the most common applications of remote sensing. The accuracy of image classifications is, however, often viewed negatively. Here, it is suggested that the approach to the evaluation of image classification accuracy typically adopted in remote sensing may often be unfair, commonly being rather harsh and misleading. It is stressed that the widely used target accuracy of 85% can be inappropriate and that the approach to accuracy assessment adopted commonly in remote sensing is pessimistically biased. Moreover, the maps produced by other communities, which are often used unquestioningly, may have a low accuracy if evaluated from the standard perspective adopted in remote sensing. A greater awareness of the problems encountered in accuracy assessment may help ensure that perceptions of classification accuracy are realistic and reduce unfair criticism of thematic maps derived from remote sensing.

Global and Local Assessment of Image Classification Quality on an Overall and Per-Class Basis without Ground Reference Data

Ground reference data are typically required to evaluate the quality of a supervised image classification analysis used to produce a thematic map from remotely sensed data. Acquiring a suitable ground data set for a rigorous assessment of classification quality can be a major challenge. An alternative approach to quality assessment is to use a model-based method such as can be achieved with a latent class analysis. Previous research has shown that the latter can provide estimates of class areal extent for a non-site specific accuracy assessment and yield estimates of producer’s accuracy which are commonly used in site-specific accuracy assessment. Here, the potential for quality assessment via a latent class analysis is extended to show that an estimate of a complete confusion matrix can be predicted which allows a suite of standard accuracy measures to be generated to indicate global quality on an overall and per-class basis. In addition, information on classification uncertainty may be used to illustrate classification quality on a per-pixel basis and hence provide local information to highlight spatial variations in classification quality. Classifications of imagery from airborne and satellite-borne sensors were used to illustrate the potential of the latent class analysis with results compared against those arising from the use of a conventional ground data set

Impacts of ignorance on the accuracy of image classification and thematic mapping

Thematic maps are often derived from remotely sensed imagery via a supervised image classification analysis. The training and testing stages of a supervised image classification may proceed ignorant of the presence of some classes in the region to be mapped. This violates the assumption of an exhaustively defined set of classes that is often made in classification analyses. In such circumstances, the overall accuracy of a thematic map produced by the application of a trained classifier will be less than the accuracy of the classification of the test set by the same classifier. This situation arises because the cases of an untrained class can normally only be commissioned into the set of trained classes. Simple mathematical relationships between classification and map accuracy are shown for assessments of overall, user's and producer's accuracy. For example, it is shown that in a simple scenario the accuracy of a thematic map is less than that of a classification, scaling as a function of the abundance of the untrained class(es). Impacts on other estimates made from thematic maps, such as class areal extent, are also briefly discussed. When using a thematic map, care is needed in interpreting and using classification accuracy assessments as sometimes they may not reflect properties of the map well